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PREFACE 



This is the third in a series of critical issues papers prepared periodically by the 
Literacy Assistance Center in New York City. Each publication presents, in 
summary fashion, information on a specific aspect of adult literacy. 
Perspectives on Assessment from the New York City Adult Literacy Initiative 
examines one of the most pressing issues facing education today. Educators in all 
systems, in every state, are searching for ways to assess more effectively the impact of 
education on students and on their learning. Further, they seek to understand which 
combination of assessment techniques will provide appropriate, meaningful and fair 
evaluation of learners' achievements. 

These quests for understanding may be of even greater importance in adult 
literacy programs, where students participate voluntarily, where their backgrounds are 
extraordinarily diverse, and where their reasons for participation are varied and 
complex. For example, some students seek to improve their employment opportunities, 
others to participate more actively in their communities and churches, others to 
expand their ability to help their children academically, and still others to gain a 
richer understanding of their new country and its systems. As literacy professionals 
strive to make basic education relevant to adult students' goals and aspirations, they 
also struggle with finding approaches to assessment that will support their educational 
efforts. 

In this paper, Deborah D'Amico-Samuels reviews the assessment process as it 
exists today in the New York City Adult Literacy Initiative (NYCALI), and explores 
a few of the changes in direction and in practice currently underway. Graphically, on 
the cover of this report, wc have attempted to portray a balanced view of different 
assessment approaches so that the strengths and weaknesses of each can be understood. 

vii 



The recommendations present challenges and directions for the future, keeping in 
mind that assessment must focus on adults' special needs, their learning goals and their 
aspirations for using enhanced literacy skills. 

We are grateful for the critical input of many practitioners in NYCAL1 programs, 
and extend our sincere thanks to Ed Noriega for his careful and thoughtful art work 
and to Cristina Di Meo for her exceptional work in preparing this manuscript for 
publication. 

The purpose of this paper, and of the entire series of critical issues papers, is to 
provide information that will enhance the understanding of adult literacy students 
and programs in New York City and lead to improved basic education services. We 
hope that the information presented here pushes the dialogue on assessment further, 
leading to a greater balance among varied approaches to assessing, understanding and 
measuring learner achievements and program outcomes. 



Karen Pearl 
Executive Director 

November 1991 
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INTRODUCTION 



Assessing the impact of adult literacy instruction is of major concern to policy- 
makers, funders, program administrators, practitioners, sid i>dult learners. 
This paper will review the national context of concern surrounding literacy 
assessment, and summarize the debate over standardized testing and other forms of 
assessment. The assumptions and conclusions of those holding different positions on 
appropriate ways to measure the progress and achievements of adult learners will be 
briefly reviewed. Data on assessment as practiced in New York City Adult Literacy 
Initiative (NYCALI) programs will then be presented. This discussion will include 
information on standardized test scores as well as descriptions of other forms of 
assessment which New York City programs use or are developing. Conclusions will 
center on recommendations for improving the quality of assessment in NYCALI 
programs, in ways which inform funders, policymakers, practitioners and students. 



PERSPECTIVES 
ON THE 
PURPOSES OF 
ASSESSMENT 



THE NATIONAL CONTEXT 



Contemporary public discourse on li teracy reflects the degree to which education 
is held responsible for many negative trends in U.S. society. Rising 
unemployment and homelessness, unacceptably high dropout rates, the 
declining status of the U.S. in the global economy, escalating drug use and violence, 
and a host of other socio-economic problems are often blamed on the failure of our 
nation's schools. Likewise, school reform or restructuring, along with adult education, 
is sometimes presented as a panacea for all of our social and economic ills. These high 
expectations for education ensure that debates over instructional theory, and by 
implication, assessment, will be politically and emotionally charged. At issue are 
differing definitions of literacy, different beliefs about effective instruction, competing 
images of adult learners and their goals and varying predictions concerning workforce 
demography and workplace literacy requirements. 



Involved in the debate over appropriate and meaningful assessment are the 
different vantage points of funders, employers, policymakers, teachers and learners. 
Politicians accountable to local, regional and national electorates need easily 
understood measures of impact and progress and clearly demonstrated outcomes. They 
may share with policymakers and funders the need to make broad comparisons among 
large groupings of students. These requirements are best addressed by standardized 
tests. However, as many funders and legislators are aware, such instruments do not 
capture the critical thinking abilities important to potential employers of adult 
learners. In addition, questions have been raised about whether standardized 
instruments are appropriate for measuring the capacities of learners as diverse as those 
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enrolled in N YCALI programs. For these and other reasons, many policymakers have 
become interested in improving such tests, combining their measures with other forms 
of assessment, or devising new ways of evaluating learning gains. 

In addition to scores which rank students according to standardized criteria, 
classroom teachers require assessment instruments that recognize the degrees of 
progress adult learners make and the variety of ways in which these are manifest. They 
need tools to assist them in planning curriculum, understanding the goals of adult 
learners, and assessing their pedagogy. Some teachers may wish to develop such tools 
for use in addition to standardized tests. Others may feel they learn very little from 
the standardized test scores of their students, and so rely on or develop a range of 
alternative methods which capture the learning progress and problems of diverse 
students. 

Program administrators may recognize these needs and welcome the development 
of alternatives to standardized tests, but have to reconcile their use with goals for 
program-wide achievement and evaluation as well as with funding requirements. 
Adult learners often need to take standardized tests for a variety of reasons, including 
job placement. Thus, they may feel frustrated if their test results do not reflect their 
efforts and achievements. They may see little relation between the test material and 
what they have 'learned in their classrooms and in their daily lives, and may be insulted 
by scores which compare their abilities to those of children. At the same time, they 
recognize that their scores on such tests are often important to employers and others. 

National, regional, program and individual learner assessment all involve attempts 
to understand the impact of adult education programs on learners' literacy. However, 
there are differences in the goals of each kind of evaluation, resulting in different 
choices and combinations of assessment methodology. 
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APPROACHES 
TO LITERACY 
AND 
ASSESSMENT 



Literacy as Autonomous Skills 

Those who view literacy as a set of skills, autonomous or independent of the context 
in which these skills are used, tend to favor instructional approaches which stress mastery 
of the mechanics of reading and wri ting (i.e., spelling, grammar, phonics, etc.) Standardized 
tests, which ask learneis to respond to questions based on a given text and to make choices 
among items based on grammatical rules, may be seen as fairly accurate measures of 
student abilities by those who see literacy in this way (Hill & Parry, 1988, pp. 1*1 1). This 
does not preclude the use of other kinds of assessment tools; portfolios of student work, 
for example, can include examples of mastery of literacy skills. 



Multiple Literacies 



Those who see literacy as embedded in socio-cultural contexts argue that adults may 
read and write for different purposes and with varying degrees of competence, depending 
on the task at hand, its meaning, and its setting (Szwed, 1981; Reder, 1988). Reading is 
seen as an active process, in which the reader interacts with the text and the context, 
drawing on his/her own experience in deriving meaning from what is written. According 
to this view, literacy practices vary with cultural and class factors. Ethnographic work in 
U.S. communities provides evidence for multiple literacies, or a variety of ways in which 
literacy can be acquired and practiced (Fishman, 1990; Heath, 1983; Lofty, 1990; Mtller 
&VanderLei, 1990; Petrosky, 1990; Taylor & DorseyGaines, 1988). Wikelund provides 
a definition derived from this conception of literacy: 

Literacy is a culturally organized system of skills and values learned and practiced 
in specific settings (Scribner & Cole, paraphrased in Wikelund, 1990, p. 5). 

For these researchers and practitioners, a meaning-centered and cxpcrienccbascd 
instructional approach allows learners to draw on their own experiences and literacy goals 
in the process of learning to read and write. Such whole language methods emphasize the 
process of making meaning through reading and writing. Concern with the mechanics of 
spelling, grammar and sound^ettcr correspondence may act as a barrier to the beginning 
reader/writer and is secondary in this approach. Approaches to assessment based on this 
conception of literacy tend to stress individualized and small group processes which value 
the role of learners in assessing their own work. These methods, often called alternative 
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assessment (meaning alternative to standardized or uniform tests) or authentic 
assessment (emphasizing the goal of information which tells you about actual reading 
behavior) are described in the section on Alternative Assessment* with examples 
given in New Directions in Learner Assessment: Examples from NYCALI programs. 

Standardized Tests 

The term "standardized" means that all test takers will be given identical 
directions, time limits and questions, and will be graded according to the same criteria. 
Such tests are usually paper and pencil multiple choice instruments (National 
Commission on Testing and Public Policy, 1990, pp. 2-3). Traditionally, scores on 
adult standardized tests in reading are translated into "grade level equivalents." Such 
equivalents imply comparability between the grade level adult learners are assigned 
based on their test scores and the performance of youngsters in the corresponding 
grades on the same tests. However, in programs where the Test of Adult Basic 
Education (TABE) is used, the grade equivalent is actually a theoretical relationship 
between the test performances of adults who take the TABE and youth who take 
standardized tests appropriate for their age. 

Standardized tests are relatively easy to administer and grade; for these reasons, 
they are cost-effective. The scores attained make it easy to compare individuals and 
programs, as well as city, state and regional groups of test-takers. The numerical scores 
yielded by these instruments are thought to be less subject to possible bias than the 
judgments of individual teachers. Because of these advantages, standardized tests are 
widely used. For example, it is estimated that 105 million such tests were administered 
in the 1986-87 school years throughout U.S. public schools (Business Council for 
Effective Literacy [BCEL], 1990, p. 1). Standardized tests are often used as critical 
indicators of student and school success, and as such are vested with considerable 
power. The same is true for the field of adult education: 

Assessment in adult literacy is a central issue with high stakes. The authority 
vested in these [standardized] tests can determine the way programs are 
developed, what is taught, and the climate of teaching and learning. It shapes 
legislation and the funding policies of public and private agencies. It is tied to 
welfare eligibility for young parents. It drives government job training programs. 
It can deny entry into the military, or crucial access to a diploma or a job 
(BCEL, 1990, p. 1). 



Other Assessment Methods 



Critical reflection by students on their own work often forms an important part 
of the assessment strategies and procedures used in this approach. Adults may be asked 
to choose some of their written work for a portfolio, and to comment on what they like 
about their selections and what they would like to improve. Or, they may participate 
with the instructor in setting goals for themselves and in periodically evaluating their 
progress. Interviews with adult learners, at intake and after a specified number of 
hours of instruction, may serve as a vehicle for assessing what adult learners want to 
be able to achieve with improved literacy and whether or not they are making progress 
toward their goals. Teacher and peer observations, student readings accompanied by 
discussion of meaning and reading strategies, performance on simulations of tasks 
important to learners in their lives, and compilation of portfolios of work are examples 
of alternatives to standardized testing (BCEL, 1990, p. 7; Lytle & Wolfe, 1990, pp. 51- 
57; Palmer Wolf, 1989; Valencia, 1990). 

While these methods are labor intensive and highly individualistic, they are also 
organically related to the content of instruction and to learners' lives and goals. They 
allow adults to evaluate themselves and each other in terms of their goals for their own 
lives and the progress they have made on tasks and skills they have identified as 
important. As such, they respond to the diversity of adult learners and the varied ways 
in which they learn. The results inform both the teacher and learner. 

The main disadvantage to alternative and authentic forms of assessment occurs at 
the level of program evaluation and policymaking, when it becomes necessary to 
compare student performances and assess learning on a broader scale. As yet, no 
methodology has been developed that allows non-standardized assessment to be 
quantified in a way that meets the need for such evaluation and research data. 



Competency or Performance-based Assessment 

One advantage of some alternatives to multiple choice standardized tests is that 
they more closely mirror how learners perform tasks requiring different kinds of 
literacy in their daily lives. Coupled with the debate surrounding the validity of 
standardized scores, this advantage has resulted in the continual refinement of 
instruments used to test large numbers of students. For example, the National 
Assessment of Educational Progress (N AEP) study done in 1985 on a sample of young 
adults used simulation tasks designed to measure a range of competencies in three 
kinds of literacy: prose, document and quantification. The purpose was to draw a more 
sophisticated picture of the abilities of young adults based on tasks which approximated 
the ways they use literacy in daily life. Drawing on the insights and practices of the 
whole language approach, the NAEP is currently adding sample portfolios to ks 
assessment of fourth and eighth graders and working on refining its simulations of 
adult contexts for its 1992 assessment of adults (Mullis & Jenkins, 1990). 

Other kinds of evaluation which build on some of the goals of authentic assessment 
while meeting the needs for more measurable data on learning progress are competency 
based assessment and the use of curricula with identifiable completion points. An 
emphasis on performance, however, can also result in highly standardized assessment 
and curricula. The California Adult Student Assessment System (CASAS), a statewide 
standardized curriculum and testing package, developed a list of competencies for 
Basic Education (BE) and English for Speakers of Other Languages (ESOL) programs 
and designed teaching and testing materials tied to these. These competencies consist 
of tasks adults might be expected to perform as consumers, workers and citizens. 
Critics of CASAS believe that when all teaching and learning activities are subsumed 
under a list of pre-identified skills, teachers and students are denied the opportunity 
to participate creatively in the learning process. They argue that a standardized list 
of competencies does not respond to the diverse needs and experiences which adult 
learners bring to the classroom, nor does it encourage the development of critical 
thinking among learners regarding the social roles they are expected to perform 
(Auerbach, 1986). Competency-based testing is often motivated by a concern for 
functional literacy, to which we now turn. 
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Employment, Policy and Functional Literacy 



Definitions of functional literacy are clearly related to concerns regarding economic 
development, national goals and employment issues. These definitions may involve 
national, international or workplace-specific standards of literacy and vary greatly 
through time. For example, Barton and Kirsch point out that a century ago, the 
literacy standard in the United States was the ability to sign one's name; fifty years 
ago, this standard made a huge jump, to a fourth grade reading level, while twenty-five 
years ago, an eighth grade reading level became the frequently used standard (1990, 
p. 2). These changing standards reflect the history of our economy as well as the 
relativistic nature of functional literacy. According to materials heralding the World 
Conference on Education, held in Thailand in 1990: 

The scope of basic learning needs and how they should be met varies with individual 
countries and cultures, and inevitably, changes with the passage of time (1990). 

In the contemporary U.S., factors affecting the economy add urgency to these 
debates over the definition of literacy, the role of adult learners in fashioning their 
own literacy, what constitutes effective instruction and how the literacy of adult 
learners should be evaluated. Studies such as the Hudson Institute's Workforce 2000 
drew attention to the changing demands of the contemporary workplace and 
demographic changes in the U.S. workforce (1987). Although there is disagreement 
over whether or not the majority of new jobs will require more educated workers, 
certainly most of the more desirable positions will. Researchers at the National 
Center on Education and the Economy argue that the decline of the U.S. manufacturing 
sector offers employers choices which have different national consequences. They 
favor rising both the skill and wage level of U.S. employees, and cite the positive 
experiences of companies who have chosen this path (1990). 

The need for more educated and technologically skilled employees converges with 
increasing proportions of minorities, women, immigrants and elderly workers in the 
nation's workforce. Members of these groups have often been excluded from the 
workplace, as well as from educational opportunities, because of discrimination based 
on race, language, age or gender. Assessment of adults for job-readiness must take into 
account the actual requirements of existing and anticipated jobs and the role of 
literacy per se in providing access to employment. 
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Functional literacy is a social and economically motivated concern with what 
adults can do in real situations and as such may draw from either of the two approaches 
to adult education discussed above. A workplace literacy project may emphasize 
reading and writing in a job-specific context, as did a Center for Employment Training 
project designed for single mothers and funded by the Rockefeller Foundation. While 
the project's approach emphasized literacy as skills, these skills were taught in the 
context in which they were used, rather than in a classroom setting. No standardized 
testing was done; rather, the success of this project was measured by the proportion of 
participants who -attained jobs and by their salary levels. While these evaluation 
measures were not standardized, they were easily quantified and performance-based 
(BCEL, 1991, pp. 6-7). Another example of performance-based assessment is the 
National Assessment of Educational Progress, a 1985 study which aimed to simulate 
adult contexts and situations in order to assess varying literacy abilities on a variety 
of tasks. 

The Report of the National Commission on Testing and Public Policy, reflecting 
a similarly pragmatic orientation, recommends the use of a variety of assessment 
techniques. The Commission bases this recommendation on the changing needs of 
our economy and social institutions, which give testing and assessment new goals. 
Standardized tests have been criticized as culturally-biased, particularly against 
minorities. Given that a significant number of new entrants to our workforce and of 
young people in our schools are from historically iisempowered social groups who 
have not fared well on standardized tests, the Commission concluded: 

Our central finding is that many current practices in education and employment 
testing stand in the way of efforts to identify and develop talent and to improve 
the functioning of key social institutions (1990, p. 5). 

Reasoning that "in part, group differences in test scores simply reflect 
social realities," the Commission staled that the use of such scores alone to 
allocate opportunities compounds existing social inequalities. A key finding of 
their research was that "no single form of assessment can shoulder the unbearable 
weight of being the sole measure of worth." While they recognize the utility of 
a standardized test as a "tool of institutional accountability, w they recommend 
that these be used in concert with "carefully crafted assessment devices [which] 
would ask students to supply answers, perform observable acts, demonstrate 
skills, create products and supply portfolios of work" (pp. 13, 27, 30). 
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The work of the Commission is important to consider alongside the debate over 
a national assessment device which would monitor progress on the national goals for 
education which emerged from the National Conference of Governors in 1990 
(Report on Education Research, 1991 ). Pointing out that the testing industry is itself 
accountable to no one, though its products are used for accountability purposes by 
institutions, the members of the Commission caution that: 

Fair test use would seem to require at the very least that the inadequacies of 
technology should not fall more heavily on the social groups already burdened 
by effects of past and present discrimination (p. 29). 

The Commission advocates changing the purpose of assessment from a gatekeeping 
function to one which facilitates, rather than restricts, opportunities for children and 
adults to demonstrate their abilities. 

Some states, such as Vermont and California, are attempting to design statewide 
assessment procedures which reconcile their accountability demands with critiques of 
standardized tests and the benefits of more qualitative assessment methods. Portfolio 
evaluation and hands-on performance, tasks in science are tv :> of the techniques used 
for assessment of children in some states. Many of the New /ork City Adult Literacy 
Initiative programs, whose assessment strategies are discussed in this report, use a mix 
of mandated standardized tests and alternative forms of assessment. 
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ASSESSMENT IN NEW YORK CITY 
ADULT LITERACY INITIATIVE 
PROGRAMS 



New York City's Adult Literacy Initiative (NYCALI) programs reflect the diversity of 
New York City's population; non-Hispanic whites are a minority and there is 
considerable ethnic and national diversity among those of African, Asian, Latin 
American and.European descent (Roberts, 1991, p. Bl). Adults in Initiative programs bring 
with them a wide range of experiences, skills, goals, and attitudes toward literacy. The 
educational and support services provided throughout the Initiative attempt to address these 
diverse needs and expectations. Thus, classes are offered in English for Speakers of Other 
Languages (ESOL), Basic Education (BE), Basic Education in the Native Language (BENL) 
and Mathematics (Math); programs offering each of these types of instruction, in turn, may be 
characterized by a variety of instructional and assessment methodologies. 

Assessment is used in NYCALI programs for purposes of initial placement at the 
appropriate instructional level, programmatic evaluation, contract and funding compliance, 
reporting and research, as well as for understanding learning progress which occurs as a result 
of receiving instruction (gain). The latter may be measured according to numerical increases 
in uniform test scores, or by other indicators of student achievement. 
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UNIFORM TESTING 
AND THE 
NEW YORK CITY 
ADULT LITERACY 
INITIATIVE 



NYCALI programs are required to report gains achieved by their students, 
as measured by two tests: the Test of Adult Basic Education (TABE) for BE 
students and the oral/aural John Test for ESOL students. At the 
recommendation of the TABE developers, the instrument is not used for 
those below the 3 .0 grade level . Adu 1 ts enrolled i n BE, BEN L and ma thema tics 
classes must be tested within their first 12 instructional hours. ESOL programs 
must administer the John test within the first 36 instructional hours. Post' 
testing generally takes place after a program cycle or semester, or after a 
specified number of hours. 1 

While the TABE has been criticized for many of the same reasons as have 
other standardized instruments, it does provide a cost-effective basic skills 
test in which all items are normed on adults (TABE, 1990). Based on an 
extensive and competitive selection process, the TABE was chosen as the 
instrument best suited to the needs of the New York City Adult Literacy 
Initiative. 

Developers of the TABE have continued efforts to make the content 
relevant to adults. These efforts have had varying success (Hill & Parry, 
1988). The 1987 version of the TABE was developed to address process and 
critical thinking skills, abilities identified as increasingly important to many 
employers and educators. 

TABE scores are reported in a variety of ways; these include grade level 
equivalents, raw scores and scale scores. Scores can also be referenced to 
group norms which reflect categories of adults who commonly take the TABE 
(such as BE/GED students, vocational and technical school students). In 
addition, scores can be used as possible predictors of General Equivalency 
Diploma scores, by comparing TABE scores of NYCALI students with the 
GED scores attained by a norming group which took both tests. The 
potential for reporting scores to students, funders and policymakers in terms 
other than grade level equivalents is an important feature of the TABE. 
Students can be given raw scores, for example, which are less insulting and 



l Ciry and State program guidelines regarding post-testing are as follows: students in classes 
which meet nine or fewer hours weekly should be post-tested at the end of each one-hundred 
instructional hours; students in classes that meet 10 or more hours per week should be post- 
tested at the end of each 200 instructional hours; students in tutorial programs should be post- 
tested after svery 50 contact hours. Contact hours represent the number of hours students 
spend in the classroom receiving instruction. If 10 students attend a 2 hour class, this would 
count as 20 contact hours. 
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more meaningful than grade level equivalents. Devising ways to make scale scores an 
acceptable and understandable way to report Initiative gains is an important task for 
programs, policymakers, funders and researchers. 

The John Test, an oral placement test for non-native speakers of English, is based 
on learners' responses in English to a series of pictures depicting a typical day in the 
life of "John." Student responses to the pictures are used to measure oral comprehension, 
ability to produce a narrative and to ask questions. In NYCALI programs, the John 
Test is also used to measure gain among students who have been in programs. An 
important consideration when using change in John Test scores in this way is that the 
test was not designed to measure gain, but only to place students in appropriate 
instructional levels. Beginning in July 1992, the John Test will be replaced by the 
New York State Placement Test for English as a Second Language (N YSPLACE) for 
ESOL placement and assessment. The advantage of the NYSPLACE is that it discerns 
the intermediate levels of English proficiency with greater precision than the John Test. 

Beginning in the fall of 1991, literacy screening was introduced in ESOL programs. 
The instrument to be used was designed by Dan Rabideau of the Literacy Assistance 
Center to provide a sense of the native language literacy abilities of students and has 
been translated for use with Spanish, Chinese and Haitian Creole speakers. Students 
are asked to fill out a personal information form which requires one word or yes/no 
responses, to read silently four short narratives of gradually increasing difficulty and 
to provide a writing sample. Thepurpose is ro collect important background information 
on students which can be used to plan appropriate curricula. 

For Spanish-speaking adults enrolling in BENL classes, the California Test of 
Basic Skills (CTBS) is used to determine the appropriate instructional level. 2 This 
test is a Spanish language adaptation of a standardized test used by some California 
school districts for grades 1 through 8. It is a multiple choice format instrument and 
is normed with reference to performance by children in these grades. There are plans 
to replace the CTBS with the Spanish Assessment of Basic Education (SABE). 
Because the SABE was developed in Spanish, rather than translated from an existing 
English instrument, it may more accurately assess the abilities of Spanish-speaking 
learners. Developers of SABE also claim it is more culturally relevant than translations 
of tests developed in English . Like the CTBS, however, the SABE was designed for use 
with children. 



2 BENL programs in languages other than Spanish choose appropriate instruments for their 
language of program instruction. 
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Gains are measured by comparing students' pre-and post-test scores on the 
previously described tests. The percentage of students post-tested is affected by when 
students enroll; those who enter a program late in the year or semester may not have 
spent sufficient time with the program to merit testing. In addition, some students 
leave programs before the required number of hours necessary for post-testing. 

Gain on the TABE was initially defined by New York City and State as an increase 
of one or more years in grade equivalents; gain on the John Test (which is scored from 
0 to 100) was defined as an increase of 20 or more points. However, because it was 
widely felt throughout the Initiative that lower test gains could indicate considerable 
improvement among adult learners, these definitions of gain were revised in 1988. 
Since that year, a .5 increase on the TABE (or a half-grade equivalent) and a 10 point 
increase on the John Test have been accepted as demonstration of gain. Regarding the 
latter, questions can be raised about the use of the John Test to measure achievement; 
as stated above, it was designed to assess students for purposes of placement. 

Table 1 shows the percentage of students for whom both pre- and post- test scores 
were available, and the percentage of students who showed gain during the first three 
years of the Initiative, when the original definitions of gain were in place. 
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TABLE 1 

STUDENTS POST-TESTED AND SHOWING GAIN 

1985 * 1987 



STUDENTS BE ESOL ALL 



FY 1985 

% of students post-tested 47.9% 59.7% 54.5% 

% of post-tested students with gain 43.8% 38.0% 41.0% 



FY 1986 

% of students post-tested 45.4% 52.7% 49.5% 

% of post-tested students with gain 43.6% 32.2% 36.9% 



FY 1987 

% of students post-tested 60.2% 68.5% 64.1% 



% of post-tested students with gain 51.2% 35.7% 42.1% 

Data source: New York City Adult Literacy Initiative Final Reports 



As indicated, the proportion of post-tested students showing gain dropped slightly 
from 41.0% in 1985 to 36.9% in 1986. Thereafter, it increased to 42.1% in 1987, the 
highest for the three fiscal years. Among BE students, 43.8% showed gain in 1985; the 
comparable figures for 1986 and 1987 were 43.6% and 51.2% respectively. However, 
for ESOL students, the highest proportion of students post-tested registered gain in 
1985, while 1986 had the lowest proportion of students with gain. 

For the 1988, 1989 and 1990 program years, the revised definition of gain was 
used. Tabic I shows the percentage of students post-tested and those who registered 
gain during these years. 
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TABLE 2 

STUDENTS POST-TESTED AND SHOWING GAIN 

1988 * 1990 



STUDENTS BE ESOL ALL 



FY 1988 

% of students post-tested 61 .0% 68.9% 64.3% 

% of post-tested students with gain 62.0% 59.8% 61.0% 



FY 1989 

% of students post-tested 61.9% 68.8% 64.6% 

% of post-tested students with gain 64.2% 58.3% 60.7% 



FY 1990 

% of students post-tested 61.7% 70.6% 65.3% 

% of post-tested students with gain 64.7% 53.2% 57.3% 



Data source: New York City Adult Literacy Initiative Final Reports 



Using the current criteria for gain, it is striking to note that the majority of 
students for whom there are pre- and post- test scores registered gain. Looking at the 
overall population (BE and ESOL students combined), 61.0% of the post-tested 
students registered gain in 1988 compared with 57.3% in 1990. However, among BE 
students, the highest proportion of post-test gain was recorded in 1990. 

Despite potential shortcomings of standardized tests, their use by Initiative 
programs has demonstrated some important baseline information (seeTables3 and4). 
The more hours students spend in instruction, the more gain they are likely to show. 
As might be expected, those whose entry test scores are low achieve the greatest gain 
(see discussion of "ceiling effect" on page 21). 
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TABLE 3 
MEAN JOHN TEST INCREASE 



ESOL STUDENTS' 


1985- 


1986- 


1987- 


1988- 


1989- 


TEST GAIN 


1986 


1987 


i no o 

1988 


1 o o c\ 

1989 


1990 


ENTRY LEVEL 












Level I 


17.4 


20.5 


20.0 


18.4 


18 7 


Level II 


13.9 


16.8 


15.7 


15.3 


14.9 


Level III 


8.6 


11.1 


11.0 


10.8 


10.5 


Level IV 


2.6 


4.0 


3.6 


5.3 


4.0 


CONTACT HOURS 












< 21 


9.9 


8.3 


10.3 


7.4 


6.6 


21 - 40 


11.9 


10.0 


10.1 


8.5 


9.8 


41 - 60 


12.5 


12.2 


12.7 


11.3 


11.9 


61 - 80 


13.2 


14.2 


13.9 


12.2 


13.5 


81 - 100 


13.1 


14.0 


14.6 


13.2 


13.7 


101 - 120 


13.1 


16.6 


15.7 


14.2 


14.6 


> 120 


14.2 


18.7 


19.4 


19.1 


17.8 



Data Source: New York City Adult Literacy Initiative Data Base 
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TABLE 4 
MEAN TABE TEST INCREASE 



BE STUDENTS' 1985- 1986- 1987- 1988- 1989- 

TEST GAIN 1986 1987 1988 1989 1990 



ENTRY LEVEL 



3 - 4.9* 


1.17 


1.16 


1.13 


1.13 


1.00 


5 - 6.9 


.74 


1.05 


1.01 


1.00 


1.01 


7 - 8.9 


.38 


.90 


.81 


.82 


.92 


)NTACT HOURS 












<21 


.66 


.54 


.57 


.93 


.76 


21 -40 


.79 


.80 


.84 


.85 


.74 


41 -60 


.81 


.86 


.87 


.91 


.88 


61 -80 


.89 


.86 


.91 


.75 


.96 


81 -100 


.87 


.90 


.95 


.85 


.88 


101 -120 


1.14 


1.04 


.86 


.84 


.96 


>120 


.92 


1.07 


1.09 


.99 


.95 



Data Source: New York City Adult Literacy Initiative Data Base 

Figures on gain are affected by a number of factors, some of them integral to 
problems with standardized tests. While tests are notorious for the anxiety they cause 
test takers, a more important consideration is how this affects test scores and 
performance. For NYCALI students, test trauma or anxiety may be compounded by 
other factors, such as prior negative experiences with testing and with the impact of 
test scores on schooling, length of time away from school and testing, or, in the case 
of some recent immigrants, little prior experience with either formal schooling or 



* NYCALI programs no longer report the test scores of students scoring below rhc 3rd grade 
level. The Initiative has no standard indicator for placement or gain for these students, as the 
producers of the TABE do not recommend it for this level. 
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standardized testing. Participation in NYCALI programs is not contingent on 
test scores; this allows instructors and program intake personnel to mitigate 
some of this anxiety by reassuring students about the purpose of standardized 
tests in their particular program. 

Because testing before placement can negatively affect students, and 
consequently, their retention, NYCALI programs are cautioned not to test 
students before they attend a class. Instead, initial placement can be based on other 
intake procedures, such as an informal reading inventory and an interview with the 
student. Some of the other negative effects of testing can be mitigated by 
practitioners as well. Scores, for example, can be reported to students and 
explained to them in terms other than grade levels. If the official test forms are 
intimidating to students, programs can experiment with their own more user^ 
friendly versions. Despite these caveats, it is likely that anxiety regarding 
standardized tests will continue to affect the numbers of students who register 
gain according to NYCALI definitions. 

Whether or not students register gain is also affected by the length of time they 
have been in a program at the time of post*testing and by their entry level scores. Test 
scores are generally subject to "ceiling effect," which refers to the artificial restriction 
imposed by tests on the range of attainable scores (Borg & Gall, 1979). For example, 
a student who answers 95 out of 100 questions correctly on a pretest can at most 
register a post'test gain of 5 points. On the other hand, a student who scored 20 in the 
same pretest can improve his/her post*test score by 80 points. This is problematic for 
data on gain which does not differentiate test takers by entry level score. 

Another issue to address concerns post*testing rates and their impact. Not all the 
students in the Initiative are available for post^testing, and there is at present no 
measure of how representative those who are post^tested are of the entire student 
population. 

In the field of adult education, regular and systematic collection of aggregate data 
on gains, particularly on a scale comparable to that of the NYCALI, is rare. Theory 
regarding the ongoing development of literacy in adults is likewise underdeveloped. 
Thus, the data on gain in NYCALI, while seemingly modest, can help to inform our 
understanding of what constitutes progress among adult learners across programs and 
regions. The use of scale scores in reporting could be used to establish more 
appropriate conventions for interpreting gain. Although scale scores, compared with 
derived scores such as grade equivalents and reference group percentile ranks, are not 
well'Suited to direct interpretation of individual performance, their equalinterval 
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property makes them very useful for a wide range of statistical analysis. Thus, scale 
scores could be very helpful in statistical comparisons for differences in test gain 
between various categories of students in the Initiative. When combined with the 
learner assessment and teacher research being done at the classroom level, this data 
can be used to shape realistic expectations for adult education programs, services and 
goals. 

While adult learners do not systematically report achievements related to their 
education to program staff, each year many thousands obtain jobs, register to vote, 
become U.S. citizens and begin to support themselves without public assistance. 
Thousands of others advance from ESOL to BE classes or from BE classes to High 
School Equivalency programs; many others attain thiir GEDs. 

In an attempt to capture some of the less easily measured aspects of the effects of 
participation in NYCALI programs, the LAC is involved in a longitudinal study of 
adult learners. During interviews for the LAC study conducted in 1988 and 1989, 
NYCALI students were asked about the impact of adult education on their lives. As 
Tables 5, 6 and 7 indicate, the great majority of students reported improvement in 
their reading, writing and (for ESOL students) English communication abihries. 
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TABLE 5 

IMPROVEMENT IN ABILITY TO SPEAK ENGLISH 



ESOL 
N = 111 


BE* 

N = 25 




N 


% 


N 


% 


DEGREE 










A lot 


31 


27.9 


18 


72.0 


A little 


70 


63.1 


6 


24.0 


Not much 


10 


9.0 


1 


4.0 


CAUSE 










Program 


83 


82.2 


18 


75.0 


Program and other factors 


10 


9.9 


3 


12.5 


Other factors 


8 


7.9 


3 


12.5 



*Question asked of non-native speakers of English who said their English had 
improved. 



Data source: Adult Learners* Perceptions of Literacy Programs and the Impact of 
Participation on Their Lives (Phase Two of a Longitudinal Study: An Interim Report) 




TABLE 6 

IMPROVEMENT IN STUDENTS' READING 



ESOL 
N = 131 


BE 

N =82 




XT 

IN 


VO 


XT 

IN 


OA 
A) 


DEGREE 










A lot 


42 


32.0 


46 


56.1 


A little 


72 


55.0 


30 


36.6 


Not much 


17 


13.0 


6 


7.3 


CAUSE 










Program 


91 


79.8 


58 


76.3 


Program and other factors 


14 


12.3 


10 


13.2 


Other factors 


9 


7.9 


8 


10.5 



Data source: Adult Learners' Perceptions of Literacy Programs and the Impact of 
Participation on Their Lives (Phase Two of a Longitudinal Study: An Interim Report) 
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TABLE 7 

IMPROVEMENT IN STUDENTS' WRITING 



ESOL 
N = 132 


BE 

N = 83 




N 


% 


N 


% 


DEGREE 










A lot 


36 


27.2 


39 


47.0 


A little 


67 


50.8 


23 


27.7 


Not much 


29 


22.0 


21 


25.3 


CAUSE 










Program 


84 


81.6 


56 


90.3 


Program and other factors 


9 


8.7 


2 


3.2 


Other factors 


10 


9.7 


4 


6.5 



Data Source: Adult Learners' Perceptions of Literacy Programs and the Impact of 
Participation on Their Lives (Phase Two of a Longitudinal Study: An Interim Report) 
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In addition to the information found in these tables regarding improved skills, 
responses to other interview questions indicated that the skills of the study participants 
had improved in ways that positively affect work performance (for employed 
interviewees) as well as ability to help children with schoolwork and to inter, .t with 
school personnel (for respondents who are parents). Increased independence and self- 
esteem, as well as the growing ability to take on the literacy demands of shopping, 
health care, transportation and community activities were also reported. Similar self- 
reported assessments have been integral to some statewide evaluations of adult 
education. Darkenwald and Valentine, for example, used interviews to gather 
self-reports about program impact for the state of New Jersey (Wolfe & Lytle, 
1989, pp. 26-27). 

For funders of NYCALI programs, assessment choices must respond to a host of 
considerations shaped by educational goals and fiscal realities. In order to present this 
perspective, City and State policymakers were interviewed for this report and their 
views are summarized here. According to John Casey, until September 1991, the 
Director of Adult Literacy for the New York City Mayor's Office of Education 
Services, the need is for "a unified system that can make statements about students 1 
learning and be used to support and justify adult education programs." Garrett 
Murphy, Director of Continuing Education Planning and Development for New York 
State, links the demand for accountability to the increase in funding for literacy and 
the need to prove the benefits of adult education to legislators and the tax-paying 
public. Because real life impacts of adult education, such as obtaining a job or getting 
a promotion, are complex outcomes affected by many factors outside the purview of 
adult education services and because they would involve costly and difficult tracking 
of individuals, academic achievements, such as test scores, are conventionally used for 
large scale assessment. 

Casey stresses the need for a mix of assessment practices, which together address 
the diverse needs of learners, practitioners, program administrators and funders. 
Accordingly, the 1991 guidelines for program funding indicate that appropriate 
evaluation and assessment, which take into account learner goals and which involve 
students as decision makers, are priorities. While recognizing the many drawbacks of 
standardized tests, Casey maintains that they do allow us to make some statements 
about learning achievements, especially regarding the different rates of gain r.^sociated 
with the varying reading levels of entering students. Such achievement standards can 
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also serve as a safeguard against marketing claims associated with commercially 
produced reading packages that promise to deliver literacy within a short, specified 
time frame. Because standardized tests are often used in our society to structure access 
to jobs, training or education, Casey asserts that adults should be prepared to take and 
interpret such tests, and this preparation is part of the job of adult education programs. 

Within the limits of standardized instruments, Casey feels that the TABE is a 
reasonably good predictor of reading ability, though the performance of an individual 
on the TABE does not "mirror the reading process." He cautions, however, that this 
predictive power does not apply as strongly to the top- and bottom-range scores. The 
TABE, because it also uses scale scores, does allow for the development of an 
alternative to reporting scores as grade levels, something Initiative programs can and 
should explore. 

Murphy, from the New York State Education Department, exposes a need for 
agreement among practitioners, administrators and funders on the goals of adult 
literacy education, which could then lead to generating an assessment package with 
a variety of tools and techniques. Key to implementation of curricula and assessment 
tailored to these goals would be staff development which allowed teachers to 
contextualizt , organize and select materials appropriate for their learners. With such 
a system place, standardized testing could be done on a sample of learners for each 
program to evaluate broad program and regional gains. However, the interest at the 
national level in a standardized test may preclude development of such alternatives. 

Assessment of beginning readers poses additional challenges for those concerned 
with adult education. Murphy reports that New York may be unique in providing 
classroom instruction for large numbers of beginning readers and writers; other states 
often rely on volunteers for these learners. This population is a mandated priority 
according to the terms of legislation on adult literacy, but there is no standardized 
instrument which can be reliably used at this level. 

All of the policymakers interviewed would like to work with programs and 
practitioners on quantifying some of the results of assessment alternatives to 
standardized tests. Russell Kratz, Chief of the Bureau of Continuing Education Field 
Services for the Division of Continuing Education Programs of the New York State 
Education Department, is particularly intrigued by the kinds of behavior changes 
adult learners might report such as "being able to write a note, help a child with 
homework, or read the TV guide instead of flipping channels to see what's on." He 
would like to see assessment which combines academic improvement as reported by 
test scores and indicators of behavioral changes reported by teachers and learners or 
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assessed in groups of peers. This would provide a more multidimensional picture of 
adult learners across programs. 

Although Initiative-wide needs mandate a standardized instrument for purposes 
of measuring achievement and guiding placement, both City and State policymakers 
recognize the criticism and limitations of such measures and the burgeoning 
development of other forms of assessment. Murphy asserts that standardized tests do 
not serve the goal of documenting individual progress well, and that judgments based 
on interactions between student and teacher should be the basis of evaluation at this 
level. Casey expresses particular interest in the work done on intake processes in a 
number of Initiative programs and in the potential for incorporating some of this work 
throughout the Initiative. Murphy points out that the external high school degree 
program provides a precedent for the use of portfolios and demonstrated competencies 
without paper and pencil testing. Funders need reporting mechanisms that capture 
the complexity and individuality of some of these alternatives and tools that can be 
used across an array of programs. Both State and City officials in adult education are 
committed to working with NYCALI practitioners in developing these. 
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Many practitioners in NYCALI programs have been developing and working with 
a variety of assessment methods and tools over the past several years. These alternatives 
to standardized tests approximate "windows on students' ways of knowing" according 
to Michael Parker, Director of the BE program at the Borough of Manhattan Community 
College of the City University of New York (CUNY). As such, they serve multiple 
purposes for learners and teachers. These include appropriate placement by program 
and instructional level; a variety of measures of development and achievement in 
reading, writing, communicating and other uses of literacy; documentation of how the 
process of learning proceeds for individuals, and communication about program 
philosophy, structure and operation. 

Assessment has been the focus of many staff development activities and special 
projects throughout the Initiative. Two issues of the Information Update (a journal 
published by the Literacy Assistance Center) have focused on assessment (March 
1987 and September 1989), and an Assessment Group of practitioners has been 
meeting monthly at the LAC for nearly two years. In addition, some recipients of 
grants and fellowships which support practitioner research have focused on assessment. 
Space does not permit full acknowledgement of the extensive and ongoing work of the 
many practitioners, students and others involved in these activities. 
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The Institute for Literacy Studies at Lehman College of CUNY has been in the 
forefront cf efforts to conceptualize and implement meaningful assessment practices 
with adult learners and adult education practitioners. Currently, Marcie Wolfe, who 
directs adult programs at the Institute, is working with a group of N YCALI practitioners 
in The Adult Educators Development Project, whose mission is "reshaping relation* 
ships between assessment and instruction." These efforts reflect some common goals 
and themes of alternative assessment development among practitioners. One such 
theme is a focus on the connections among assessment, instruction and staff 
development through systematic observation and reflection by teachers. Teacher 
inquiry, combined with discussion and feedback from practitioners and students, lays 
the basis for assessment grounded in classroom practice and student experiences. This 
emphasis on teachers as researchers widens the search for appropriate assessment to 
encompass issues of pedagogy and epistemology as well; teachers observe and document 
the learning process as it intersects with the diverse learning histories, goals, 
expectations and abilities of adult learners. 

Below, the experiences of a few Initiative programs which are designing and using 
alternative assessment methods are highlighted. 

Centers for Reading and Writing, New York Public Library 

Student assessment has been an ongoing concern at the Centers for Reading and 
Writing. Both staff and students experienced frustration with the inability of 
standardized instruments to reflect growth in students 1 reading and writing, and 
Center staff shared the conviction that assessment should be an integral part of 
instruction, not an added component. Consequently, a small group assessment model 
consistent with the program's whole language instructional philosophy was developed. 
Presently, these assessments are being conducted with tutorial groups; however, Center 
staff are adapting this approach for use in the Centers' ABE and Saturday writing 
classes. 

The small group assessment model is essentially a focused and reflective discussion 
which includes a group of students who work together regularly, their tutor and a 
member of the full-time program staff. A preliminary discussion between the staff 
member and the tutorial group prepares students for the assessment. Students' goals 
recorded by the staff during the intake process, along with students' reading and 
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writing folders, are the subjects of a structured set of questions for individual students 
and the group. The assessment takes about two hours. 

According to Center staff, the model's reflective nature reinforces their learner- 
centered and process-oriented approach to reading and writing. Because the model 
draws on the observations of the tutorial groups as well as of the individual, it is 
consistent with their efforts to foster a sense of shared community among program 
participants. Center staff also provided for student input into the development of the 
model; they were asked to comment on discussion questions after reading verbatim 
transcripts of the initial small group assessments. 

Students bring their reading and writing folders to the assessment discussion. The 
folders contain all of the students' work, rather than a selection. Questions focus on 
the kinds of things people have read and written since they started: the changes they, 
their tutors and the group of learners perceive; whether or not reading and writing 
goals have been reached; what is easy or hard about the work students are doing, etc. 
One set of questions asks students to discuss how the group has helped members and 
how they can work together to meet individual and group objectives. The staff 
member conducting the assessment discussion provides a written summary to all 
participants for additions and comments, and the final written summary is analyzed by 
students and teachers and added to students' folders. 

The small group assessments provide insight into how adult learners perceive 
their progress, and how effective instruction is in helping them meet their goals. 
Students feel a sense of accomplishment and validation about their documented effort 
and take more responsibility for what they want to learn. In add. 1, the small group 
format of the discussion reinforces the sense that member} have of being part of a 
community of learners. 

While the students and staff are clear about the benefits of this kind of assessment, 
Center staff are wrestling with how to make this accessible and meaningful to funders 
and others interested in adult education, how often to conduct the assessments and 
how to reflect the changes adults report that are occurring outside the classroom 
setting. Often, learners' most convincing and significant evidence of change are the 
new things they can do at work, at home or in the community. Roger Dovner, Director 
of the Centers for Reading and Writing, is working with staff on ways to make these 
changes a larger part of the Center's assessment model. 

Currently, a group of practitioners from the Centers meets weekly to develop the 
next phase of this model. This phase involves students in sifting through the contents 
of reading and writing folders and selecting significant pieces of work for inclusion in 
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portfolios. Selection criteria depend on the individual students' decisions about what 
is important to their work, and assessment discussions with others include analyses of 
these criteria. Students are also encouraged to include significant indicators of 
progress from their daily lives in their portfolios. Several Centers have already piloted 
this new phase of the assessment model. 

York Learning Center of York College (City University of New York) 

Adult educators at York Learning Center refer to the assessment strategies they 
are developing as "authentic or appropriate" assessment, reflecting their emphasis on 
what actually occurs when students work on their reading and writing. Their approach 
to assessment includes a comprehensive intake interview and portfolio assembly by 
students. Assessment is an integral part of the curriculum, which is characterized by 
a whole language approach to an experience-based curriculum. Both assessment and 
instruction are participatory and learner-centered. 

At the initial intake interview, students choose their own reading materials from 
a selection of pieces at various levels of difficulty. The interview is designed to allow 
learners to demonstrate what they already know, and is conducted by a counselor with 
an individual student. The aim is to acquire information which will be useful for 
instruction. Questions probe for learners' attitudes, expectations, and assumptions 
about education, as well as for students' interests, educational history and literacy 
goals. Students also select writing topics and at the end of the interview, decide on 
their levels for instruction. 

The information provided to teachers assists them in selecting curricular materials 
and in making decisions about classroom activities. For example, one class at York 
Learning Center, composed primarily of students from the English-speaking Caribbean, 
did considerable work with a song by Jamaican reggae singer Bob Marley. Using the 
song as a basis for reading, writing, vocabulary building, social history and critical 
interpretation, a sense of mastery was acquired. Such classroom activities produce the 
folders of materials from which students select pieces for their portfolios. 

Portfolios consist of a selection of materials which demonstrate progress and 
change. A York Learning Center student's portfolio typically contains samples of 
student writings, reading logs and a series of checklists which record the instructor's 
observations of behaviors and accomplishments which demonstrate change. For 
example, the checklist related to "ownership of reading" asks whether students are 
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developing their own preferences regarding reading material, whether they increasingly 
share information about what they are reading, etc. Another checklist focuses on 
students' abilities to assess their own learning, and a third type aims at providing a 
window on changes in students' participation in Learning Center and classroom 
activities, such as sharing their writing with others. Together with other entries in the 
portfolio, these provide a holistic way of looking at student progress. Over time, 
improvement in skills and increased ability in written expression become evident. 
The portfolio is also the basis for student self- assessment; students are asked the 
reasons for their selections, as well as for their thoughts on their own progress and 
problem areas. 

According to Jane MacKillop, Director of the York Learning Center, portfolio 
assessment grows directly out a desire to link assessment with instruction. Portfolios 
can respond to the view which educators have of learning progress. To design their 
portfolio assessment methods, the staff at York generated lists of the ways they 
recognized progress and then looked at which of these could be documented in 
portfolios and how. 

Students from York Learning Center have developed a student newspaper, Amazing 
News, whose production grew out of their discussions and writings about the Daily News 
strike in 1990*91. In addition to the selection and editing skills students have 
practiced in the compilation of their portfolios, they are learning newspaper layout 
and design, as well as the scheduling, coordination and organizational aspects of 
publishing. A student who served as guest editor spoke of how he chose his portfolio 
essays and poetry, contrasting his satisfaction with the progress he saw in these 
writings to his negative experiences with standardized tests. 

Like many practitioners and program personnel, staff at York are committed to 
their alternative assessment methods, and share with students the satisfaction of 
documenting the progress they observe in the classroom. At issue are ways to make the 
information generated through portfolios accessible to funders and others concerned 
with achievement at a program or citywide level. 
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The New York City Public Schools 



Exploring assessment alternatives for beginning BE students has been a focus of 
both staff development and classroom activities in some New York City Public 
Schools programs. Since September of 1990, Betty Gottfried, a staff developer and BE 
teacher, has been working with groups of practitioners interested in assessment for 
beginning readers and writers. 

As discussed above, the TABE test is not appropriate for adults whose skills are 
below the third grade level. Program teachers and staff decide whether to test entering 
students based on a number of factors. Some may administer the locator test designed 
for use with the TABE, others may interview students regarding their facility with 
print. Students may be asked to read passages at a beginning level so that practitioners 
can observe their reading, or to write their names and addresses on program forms. 
Instructors may decide that the TABE test would be inappropriate based on their 
observations of these activities. Alternatively, staff may administer tests designed for 
beginning readers, such as the Slosson Oral Reading Test (SORT). 

Teachers of beginning BE students have been working with alternatives to stan- 
dardized tests which link assessment with instruction. In their staff development 
sessions, teachers suggested and refined qtestions and categories to be included in an 
intake interview. The results were similar to the areas covered in intake interviews 
discussed above; the learner's educational history, goals, expectations, and motivation 
are addressed. Sample interviews with students have been videotaped for teachers 
interested in viewing how they are conducted. 

Portfolio planning, development and use as an assessment tool has also been a 
focus of teacher development and classroom practice. A particularly useful item for 
the portfolios of beginning readers and writers is the monthly calendar. Students 
make the calendars themselves; for some these provide a useful introduction to the 
conventions of written discourse and of print. The notion of recording time in this 
way, as well as the vertical and horizontal reading of days and dates embedded in a 
calendar format are new ideas for some learners. Minimally, beginning students can 
keep a record of their attendance. As they gain facility with reading and writing, they 
can note what they have read, allowing the calendar to serve as a reading log. They 
can also indicate when they have wri ttcn in their dialogue journals. The calendars can 
thus provide tangible evidence of effort and of the quantity of work done. Over time, 
they document the increased ability of students and can be used to make connections 
among attendance, classroom activities and changes in reading and writing. Portfolios 
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also include samples of student writing and occasional narratives, by both student and 
teacher, about the student's reading and writing progress. 

Writing as an assessment tool for beginning readers and writers is also being 
explored through the use of dialogue journals. Learners write in these during each 
class meeting, and the teacher reads the entries and responds. The dialogue journal 
provides opportunities for instruction as well as for assessment. Teachers can see what 
discrepancies there are in the learner's use of language for different purposes (i.e., 
speaking, writing and reading). For example, journals allow teachers to perceive and 
discuss with students how ideas expressed in writing are sequenced, or what repeated 
difficulties arise in their writing. 

As a participant in the Adult Educators Development Project at the Institute for 
Literacy Studies at Lehman College, described above, Gottfried is currently focusing 
for this project on student and teacher perceptions of learning progress. Working with 
an instructional facilitator in one district, she is researching the form and content of 
students' oral and written reflections on their learning. The purpose is to look for 
congruence as well as disparity between teachers' expectations for student self- 
assessment and students' expressions of what they view as progress. Gottfried is also 
interested in the impact of adult education on students' lives outside the classroom, 
and is seeking ways to bring these changes into the assessment process. 

Some teachers of beginning BE students express a need for a more formal 
assessment which would serve as a companion to portfolios and student interviews. 
Such an instrument is particularly important to beginning teachers who want a more 
structured way of understanding the abilities and needs of beginning readers and 
writers. Criteria for a useful assessment tool to serve this purpose, according to 
Gottfried, include reading passages for beginning readers with adult content, and 
instructions for miscue analysis which do not overwhelm teachers with logistics for 
recording. 

Development of assessment tools for beginning readers and writers which respond 
to their particular needs as they begin to work with print materials continues to be the 
focus of a group of teachers in New York City Public Schools programs. Their work 
will provide important additions to assessment methods currently being used and 
developed by other Initiative practitioners. 
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Literacy Volunteers of New York City 



Literacy Volunteers of New York City is a community-based organization 
under the aegis of the Community Development Agency of New York City. 
Services are provided in nine learning centers, where teaching is done by 
trained volunteers who are supervised by the center director. Intake is handled 
centrally through the Pre-Enrollment Program. Jean Fargo, Associate Director 
of Education-Evaluation, describes alternative assessment as being continually 
assessed by program staff, as they work with a model which has been evolving 
over the past two years. This model was recently the subject of an external 
evaluation which included writing assessment practices and the use of portfolios; 
the results are currently available through Literacy Volunteers. 

Alternative assessment at Literacy Volunteers has two components. One 
component is designed to capture students' perception of their own progress and is 
based on a series of interviews with students over time. The other focuses on 
compiling data on students' actual reading performance, and compares students 1 
efforts at different points in time. 

The first interview of the series is conducted during the Pre-Enrollment Frogram. 
This is done by a current student working with a staff member and a trained volunteer. 
Together, they ask a group of two to four new students a series of questions and record 
each student's answers. These focus on learning goals, special interests, motivation 
for seeking instruction, educational history, and current job and family situation. 
Students are also asked what things teachers should do to help them learn, and 
whether there are any things teachers should avoid doing. In addition to acquiring 
information about the student, the interview aims to introduce students to formulating 
their own learning goals and assessing their progress toward achieving them. 

SuHequent interviews are conducted approximately every six months, and provide 
documentation of learning for students and their tutors. These interviews offer an 
opportunity for students to examine their own work in reading and writing and the 
ways in which program participation has affected their lives. Students discuss their 
immediate reading and writing goals as well as their long-term expectations. During 
the interview, students' perceptions of their work and accomplishments are 
docu mented. Students refer to their notebooks and other records of their reading and 
writing during the course of the interview. At the end of the interview, students 
discuss their goals for the next few months. During this process, students, rather than 



staff or tutors, become the observers and experts on their own learning. 

The other component of assessment at Literacy Volunteers is a reading 
documentation and assessment by a staff member or a trained volunteer. These are 
conducted at least once each fiscal year. Using a text the student is currently reading, 
the interviewer asks questions related to the choice of the material, how long the 
student has been reading it, the content of the text, etc. The student is then asked to 
read aloud, beginning from wherever s/he last finished. The interviewer records the 
student's miscues during the oral reading, and then asks the student to read the same 
passage silently. Questions focused on comprehension are then asked. The student 
is also asked to decode words from the same reading, this time presented out of context. 
The interview also includes questions for the student, the student's tutor and the 
center director regarding perception of changes in the student's reading. The reading 
documentation process concludes with a discussion among these people regarding the 
student's reading strategies and recommendations for future reading work. 

The two components of assessment at Literacy Volunteers described above have 
met their goals of generating rich information and documentation regarding the 
learning changes taking place for individuals, as well as the aim of communicating the 
importance of the student's role in learning. The resulting information on how 
students see change and evaluate instruction allow staff and tutors to discover 
developments missed during the process of instruction. 
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DISCUSSION AND RECOMMENDATIONS 



Despite philosophical variety in approaches to literacy, most of those 
concerned with adult education share some basic hopes and concerns. 
Most desire that a greater number of adults enjoy a wider range of life 
choices and educational and workplace opportunities than is currently the 
case. These shared goals depend in part on more effective instruction, which is 
what assessment aims to define and inform. 

Dialogue which involves funders, program administrators, practitioners, students 
and others is needed to determine what kinds of information assessment should 
provide and the best means for acquiring these data. Practitioners working with 
alternative assessment express interest In meeting with others involved in adult 
education to discuss these mutual concerns. Professional educational evaluators with 
experience in a variety of assessment types may be able to devise ways to quantify 
meaningfully and present information on individual learner assessment for those who 
need to place such data in a programmatic or broader context. Researchers who have 
analyzed and documented the social contexts in which adults practice literacy skills 
can bring to this conversation knowledge and methods for understanding and 
documenting the effects of program participation beyond the boundaries of the 
classroom. Because the forces that drive adult literacy are ultimately concerned with 
its impact in the workplace, family and community, measures of these must be part of 
assessment planning. Employers and service providers should be part of this 
conversation, because they are implicated in the goals of both instruction and 
assessment, especially at this time of greater cooperation between City and State 
agencies and between the public and private sectors. For example, employers can be 
sensitized to the possibilities of alternative assessment for providing profiles of 
potential employees. 
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Recommendation: 



Recommendation: 



Recommendation: 



• Forums, seminars and ongoing group discussions involving educators, adult literacy 
practitioners, researchers, adult learners, flinders, policymakers, employers, and social 
service providers should be convened to considerand make decisions jointly regarding 
purposes and methods of assessment in adult literacy. 

Within NYCALI programs, both standardized and alternative assessment have 
provided useful information. The administration of uniform tests, such as theTABE, 
CTBS, John and NYSPLACE allows for broad comparisons of gain and entry level 
scores to be made across programs. These data provide information regarding who is 
served by the Initiative, what the impact of instruction is on learners at different levels 
and the effects of varying amounts of instruction. .However, questions regarding the 
utility and cost-effectiveness of testing large numbers of adult learners at the intervals 
specified for post- testing can be raised. 

Gregory Anrig, the President of the Educational Testing Service, has publicly 
expressed a conviction that "it isn't necessary to test every child to know how children 
are doing." While recognizing the need for accountability testing where public 
funding is involved, he recommends considering cost-effectiveness and efficiency in 
making assessment decisions, as well as paying more attention to assessment which 
informs instruction and which is "very different from standardized testing." He 
cautioned that constant achievement testing uses up time that could be better spent 
on instruction (Report on Literacy Programs, 1990, p. 186). Given current fiscal 
constraints and priorities, Anrig's advice is especially pertinent. 

• Policymakers, funders, practitioners and program administrators may wish to 
discuss exactly what their goals are for standardized testing and whether or not 
these could be met by testing samples of adults enrolled in NYCALI programs. 

• Standardized testing with a sample of students might be combined with other 
assessment methods to gather more complete and more varied data on this 
selected group of learners. The results could provide a richer basis for policy 
than the information on gains currently offers. However, thought must be 
given to how sampling will be explained to students, and how being selected for 
sampling would affect students who are tested. Also, ways to meet the needs 
for program accountability and research on students at different levels of 
ability which Initiative-wide testing now serves would have to be devised. 
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Standardized tests, like other forms of assessment, are continually revised and 
updated. For example, the Educational Testing Service (ETS) has designed a new set 
of tests that "incorporate extensive use of open-ended tasks that simulate the use of 
literacy skills in a variety of relevant adult contexts" (Simon & Schuster, 1991). 
Linked to the adult literacy frameworks used by the NAEP survey of young adults, 
these tests measure document literacy, quantitative literacy and prose literacy, and 
generate scale scores rather than grade levels. While new instruments such as the ETS 
Tests of Applied Literacy Skills aim to provide a more accurate and sensitive picture 
of adult learners, they must continue to be critically evaluated for applicability, cost 
effectiveness and actual advantages in comparison to tests now in use. 

Those administering standardized tests can also consider doing so in ways that 
allow learners to focus more on what they can do and what they know and less on the 
test itself. Permitting students to ask questions during the testing process or relaxing 
the time allotted for the test may facilitate this. 

While standardized testing seems to pose a clear-cut and simple way to establish 
literacy standards for a complex population in a complicated urban environment, we 
must continue to ask what the results of these tests do and don't tell us and to search 
for alternatives which provide additional information. Given the host of questions 
raised about the utility of standardized tests, particularly for the kinds of students 
served by the Initiative, NYCALI programs can benefit from the continued search for 
more varied and accurate cessment strategies. 

Many of these strategies provide a ritual analogous to testing, but with a different 
focus and process for acquiring information about learning and achievement. Again, 
the question is whether this kind of individual learner assessment can provide 
information that is meaningful for program evaluation. If so, how? Should the kinds 
of documentation generated be summarized, quantified or analyzed for patterns? 
Should the focus be on refining knowledge about how adults and instructors define and 
recognize changes in reading and writing, or on evaluating and measuring these 
changes? If evaluation is a goal, how do values get assigned to what learners do, and 
do these reflect the values of funders, learners, practitioners and/or others? 

• The development of alternatives to standardized tests has been proceeding in 
Initiative programs over the past several years, and it may be time to ask 
programs working in this area to report on what they are doing and what they 
have learned. The resulting information can be disseminated throughout the 
Initiative in a systematic way, allowing practitioners and learners to choose 
among a variety of methods. 
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Recommendation: 



Recommendation: 



• Because of the diversity of programs and adult students in the Initiative, adopting 
specified alternative measures for use across New York City programs may be 
inappropriate and premature. Rather, programs should be given assessment choices, 
and encouraged to report their findings along with other program data. 

• Program reports can summarize, in ways appropriate to the assessment strategies 
they have been using, what they have learned from assessment. This will clarify the 
relation between program evaluation and learner assessment and facilitate the 
development of both throughout the Initiative. Programs will have the opportunity 
to provide funders with indicators of their work beyond statistics on enrollment, 
retention and test scores. 



Ultimately, it is improvement in the quality of instruction which will make the kinds 
of differences students, practitioners, policymakers and funders want to see. Because the 
theory of adult learning is still developing, assessment which is closely tied to instruction 
and which documents the learning processes of diverse adults as readers and writers is 
essential to understanding what constitutes effective instruction for adults. 

The National Commission on Testing and Public Policy entitled its 1990 report From 
Gatekeeper to Gateway, emphasizing their conviction that the purpose of assessment needs 
to be transformed from a method for evaluation and selection to a tool for development and 
opportunity. This interpretation of purpose is ultimately what is at stake in the debate 
over assessment, in which adult learners in New York City have much to gain and much 
to lose. The words of educator Mike Rose, regarding the testing of children, are equally 
germane to adult literacy assessment: 



In the name of excellence, we test and measure them — as individuals, as a group — 
and we rejoice or despair over the results. The sad thing is that though we strain 
to see, we miss so much. All students cringe under the scrutiny, but those most 
harshly affected, least successful in the competition, possess some of our greatest 
unperceived riches (Rose, 1990, p. xi). 



New York City's current economic and social crises demand that we utilize our 
resources effectively to develop the rich and varied talents of all New Yorkers; the range 
of assessment practices in N YCALI programs offers hope and direction for these efforts, as 
well as opportunity and growth for educators and learners. 
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